Something that brings pleasure but is considered taboo, unadvisable or lowbrow.
For the renowned sushi chef, eating fish sticks drowned in tartar sauce was a guilty pleasure.